feat: LLM discoverability sync from kurrentschrift - #10808

Summary

• Assistants can find the machine guide from wherever they land. The sister project kurrentschrift learned from two assistant protocols (2026-08-28, its PRs [box-basic] highcharts implementation #439/[area-basic] letsplot implementation #440) that assistants' fetch tools often allow only URLs that already appeared verbatim in fetched content — a relative link, a …-elided path or the bare /llms.txt convention leaves the API unreachable. So the guide and one complete, callable example per surface now stand at every place an agent actually reads: the SPA shell (<link rel="alternate" href="/llms.txt"> + a <noscript> block with absolute URLs — the fallback for every client the nginx map does not recognise), both robots.txt files (two terse lines at the very top of the site file, a block beside Sitemap:, the API host's comment head), every bot page's nav, and every implementation page (its own code URL spelled out + a visible <pre><code class="language-json"> retrieval record beside the JSON-LD, because HTML-to-Markdown converters drop <script> and keep <pre>). Guessed paths resolve instead of failing: /.well-known/llms.txt → 302 (it soft-404'd with the shell, verified live), api.anyplot.ai/llms.txt → 302 to the site file (it 404'd).
• Two correctness fixes. (1) bot_fetch on the prerender path was very likely mostly dropped by Plausible: on the hop Cloud Run app → Cloudflare → API, cf-connecting-ip is the app container's Google egress address — exactly the "hosting provider IP" Plausible discards — and visitor_ip preferred it over the forwarded list where the crawler stands. kurrentschrift measured one counted read in twenty with probe events; anyplot has the same topology. visitor_ip now reads the leftmost valid x-forwarded-for first, and @seo_proxy forwards the crawler explicitly. (2) The use=reference Content-Signal token was invalid (vocabulary is exactly search/ai-input/ai-train; a strict parser may discard the whole line) — dropped from both groups.
• Shared-by-decision blocks stay identical. The nginx $is_bot map gains ~*xai (a xAI-Bot UA got the empty shell — verified live) with kurrentschrift's exact comment, so the block stays byte-identical (checked with diff against kurrentschrift origin/main); AI_AGENTS gains the bare grok and xai tokens the map already served but never counted (needs mirroring in kurrentschrift — Todoist task filed). New asset_fetch event on bots.anyplot.ai records which implementation's code / which spec's detail assistants fetch through the API (cache misses — those routes stay edge-cached for the SPA; documented in plausible.md). Guards: new app/src/routes/seoCoverage.test.ts pins llms.txt ↔ route registry ↔ robots.txt ↔ shell (llms.txt gains the missing /map line it surfaced); bot-serving-check adds the .well-known redirect, robots/sitemap bypasses, the site card, a real 404, xAI-Bot and a deep-route SPA control. Doctrine: docs/reference/seo.md "Discoverability for assistants".

Plan

N/A — assessment in session; source findings: kurrentschrift docs/reference/crawler-richtlinie.md §3 "Auffindbarkeit" and its PRs #428, #435, #439, #440.

Test plan

• uv run pytest tests/unit tests/integration — 1713 + 67 passed; ruff check/format --check clean; mypy api core clean
• cd app && yarn lint && yarn fm:check && yarn type-check && yarn test && yarn build — 623 tests, build clean; dist/index.html carries the head link and the <noscript> block with absolute URLs
• Local API smoke (uvicorn against the shared DB, reads only): /robots.txt carries Content-Signal + pointers, /llms.txt → 302 https://anyplot.ai/llms.txt, bot nav ends with llms.txt · llms-full.txt, /seo-proxy/box-basic/python/plotly renders the retrieval record with real render/interactive URLs and quality_score
• app/public/robots.txt parsed with urllib.robotparser: Bytespider denied everywhere, /debug and /interactive denied for *, /llms.txt allowed — the two new top comment lines change nothing
• bot-serving-check.yml run block extracted and executed locally against the Cloud Run origin: 29 checks OK; exactly the two checks that need the new nginx config (xAI-Bot → prerender, .well-known/llms.txt → redirect) fail until the app deploy lands — expected: the daily run will be red until the frontend Cloud Build after merge, then green
• After merge: watch the app and API Cloud Builds, then curl -sI https://anyplot.ai/.well-known/llms.txt (302 → /llms.txt), curl -s https://api.anyplot.ai/robots.txt (pointer lines), curl -s -A "xAI-Bot/1.0" https://anyplot.ai/scatter-basic | grep -o '<title>[^<]*' (per-route title), and dispatch Bot Serving Check once by hand
• After merge: register the asset_fetch goal and the asset, spec, library properties on bots.anyplot.ai in Plausible (Todoist task); watch whether the 06:23 UTC bot-serving run now shows ~25 bot_fetch events on the bot site (that is the live proof of the visitor-IP fix)